apps/web/src/app/[locale]/(dashboard)/[wsId]/crawlers/(default)/page.tsx (1)

44-54: 💡 Verification agent

🧩 Analysis chain

Verify the impact of removing data transformation.

The data transformation step that added the href property has been removed. Please ensure that this property is not required by the table columns or any other components.

Run the following script to check if any code relies on the href property:

---

🏁 Script executed:

#!/bin/bash
# Description: Check for usage of the `href` property in table columns and related components.

# Test: Search for `href` usage in column definitions
rg -A 5 'href' apps/web/src/app/\[locale\]/\(dashboard\)/\[wsId\]/crawlers/columns.ts

# Test: Search for `href` usage in components that might use the table data
rg -A 5 'href' apps/web/src/app/\[locale\]/\(dashboard\)/\[wsId\]/crawlers/

Length of output: 3378

---

Issue Identified: Missing href Property for Row Actions

The RowActions component in the crawlers module expects a href property (accessed as row.original.href). Our search confirms that while one cell in columns.tsx computes a URL using row.getValue('id'), the RowActions component still relies on a href property that would have been added in the removed data transformation. This discrepancy can cause the view button (or any link relying on the href) in the row actions not to render or work as intended.

• RowActions Usage: In apps/web/src/app/[locale]/(dashboard)/[wsId]/crawlers/row-actions.tsx, the link is conditionally rendered only if href is provided.
• Computed URL: Although columns.tsx computes a URL for one of its columns, it passes row.original.href to RowActions, which will be undefined after the transformation removal.

Please verify whether this behavior is acceptable or if the code should either (a) revert the removal to ensure the href property is added to each data row or (b) update the RowActions usage to compute the href similar to the approach in the column definition.

apps/db/supabase/migrations/20250210132933_new_migration.sql (1)

45-103: ⚠️ Potential issue

Review security implications of broad permissions.

The migration grants full permissions (including truncate) to anonymous users. Consider restricting permissions based on actual requirements.

Consider removing unnecessary permissions, especially for anonymous users:

-grant truncate on table "public"."workspace_crawlers" to "anon";
-grant trigger on table "public"."workspace_crawlers" to "anon";
-grant references on table "public"."workspace_crawlers" to "anon";

apps/web/src/app/api/v1/workspaces/[wsId]/crawl/route.ts (3)

11-28: Validate the URL input more thoroughly
Currently, the code only checks for the existence of url, but doesn't validate whether it's a valid or properly formatted URL. Consider using a stricter validation step (e.g., checking for valid protocols) to prevent passing malformed URLs to the crawler.

---

86-92: Use structured logging for errors
The code logs errors using console.log. In production, a structured logging approach or a centralized logging service is preferable to simplify troubleshooting and to preserve sensitive details securely.

---

94-155: Check for missing or invalid SCRAPER_URL in environment
The code fetches process.env.SCRAPER_URL without verifying its presence or validity. If SCRAPER_URL is not set, it may produce an unhelpful runtime error. Consider returning a 500 status with a clear message if this environment variable is missing.

apps/web/src/app/[locale]/(dashboard)/[wsId]/crawlers/[crawlerId]/crawl-button.tsx (2)

6-14: Ensure consistent error handling or user feedback
The component logs errors to the console but does not provide user-visible feedback. For a better user experience, consider adding a notification/toast or UI indicator if crawling fails.

---

17-46: Handle non-OK responses more robustly
Currently, the code simply logs an error if res.ok is false and then returns. If the server returns a 400 or 403, it might be helpful to parse the JSON body to show a user-friendly message explaining why the crawl failed.

apps/web/src/app/[locale]/(dashboard)/[wsId]/crawlers/[crawlerId]/utils.ts (1)

20-53: Handle self-closing tags and complex HTML structures in formatHTML
The current approach incrementally adjusts depth and relies on </> detection. Self-closing tags, inline elements, or malformed HTML can disrupt the formatted output. Consider leveraging a proper HTML parser for a more reliable formatting approach if your use case demands it.

apps/web/src/app/api/v1/workspaces/[wsId]/crawlers/status/route.ts (1)

15-34: Enhance error handling with detailed messages.

Include the error details in the response for better debugging while ensuring no sensitive information is exposed.

   if (crawledUrlError) {
     return NextResponse.json(
-      { message: 'Error fetching crawled URL' },
+      {
+        message: 'Error fetching crawled URL',
+        error: crawledUrlError.message || 'Unknown error'
+      },
       { status: 500 }
     );
   }

apps/web/src/app/[locale]/(dashboard)/[wsId]/crawlers/(default)/page.tsx (1)

59-101: Implement search functionality.

The getData function accepts a search query parameter q but doesn't utilize it in the query builder. Consider implementing search functionality to filter the crawlers based on the query.

Apply this diff to add search functionality:

 const queryBuilder = supabase
   .from('workspace_crawlers')
   .select('*')
   .order('created_at', { ascending: true })
   .eq('ws_id', wsId);

+if (q) {
+  queryBuilder.ilike('url', `%${q}%`);
+}

 if (page && pageSize) {

apps/web/src/app/[locale]/(dashboard)/[wsId]/crawlers/[crawlerId]/crawler-content.tsx (1)

142-177: Enhance URL display with truncation and copy functionality.

Long URLs may overflow the container, and there's no easy way to copy them.

Apply this diff to improve the URL display:

+'use client';

 import CrawlButton from './crawl-button';
 import { formatHTML, unescapeMarkdownString } from './utils';
 import { MemoizedReactMarkdown } from '@/components/markdown';
+import { Copy } from 'lucide-react';
+import { toast } from 'sonner';

// ...

                              <span
                                className={
                                  relatedUrl.skipped
                                    ? 'text-muted-foreground'
                                    : ''
                                }
                              >
-                                {relatedUrl.url}
+                                <div className="flex items-center gap-2">
+                                  <span className="truncate max-w-[500px]" title={relatedUrl.url}>
+                                    {relatedUrl.url}
+                                  </span>
+                                  <button
+                                    onClick={() => {
+                                      navigator.clipboard.writeText(relatedUrl.url);
+                                      toast.success('URL copied to clipboard');
+                                    }}
+                                    className="p-1 hover:bg-muted rounded-md"
+                                  >
+                                    <Copy className="h-4 w-4" />
+                                  </button>
+                                </div>
                              </span>

apps/db/supabase/migrations/20250210132933_new_migration.sql (1)

129-134: Consider adding indexes for URL lookups.

The crawled_urls and crawled_url_next_urls tables will likely be queried by URL frequently. Consider adding indexes to improve query performance.

create table "public"."crawled_urls" (
    "id" uuid not null default gen_random_uuid(),
    "url" text not null,
    "html" text,
    "markdown" text,
    "created_at" timestamp with time zone not null default now()
);
+CREATE INDEX idx_crawled_urls_url ON public.crawled_urls USING btree (url);

create table "public"."crawled_url_next_urls" (
    "origin_id" uuid not null default gen_random_uuid(),
    "url" text not null,
    "skipped" boolean not null,
    "created_at" timestamp with time zone not null default now()
);
+CREATE INDEX idx_crawled_url_next_urls_url ON public.crawled_url_next_urls USING btree (url);

Also applies to: 136-142

.github/workflows/check-and-bump-versions.yaml (2)

296-310: Improve code maintainability.

While the logic is correct, consider these improvements for better maintainability:

1. Extract magic number "1" as a constant
2. Store "package.json" as a variable
3. Reduce nesting by early returns

Apply this diff to improve the code:

+                    # Constants
+                    PACKAGE_JSON="package.json"
+                    SINGLE_FILE=1
+
                     # If only package.json changed, check if it's only version changes
-                    if [ "$(echo "$SOURCE_FILES_CHANGED" | wc -l)" -eq 1 ] && [ "$(echo "$SOURCE_FILES_CHANGED" | grep "package.json$")" ]; then
-                      if check_package_json_changes; then
-                        echo "Only version field changed in package.json, skipping version bump"
-                        SHOULD_BUMP=false
-                      else
-                        echo "Package.json has meaningful changes"
-                        SHOULD_BUMP=true
-                      fi
-                    else
-                      echo "Source files changed:"
-                      echo "$SOURCE_FILES_CHANGED"
-                      SHOULD_BUMP=true
-                    fi
+                    # Set default to true for source changes
+                    SHOULD_BUMP=true
+                    
+                    # Check if only package.json changed
+                    if [ "$(echo "$SOURCE_FILES_CHANGED" | wc -l)" -eq $SINGLE_FILE ] && \
+                       [ "$(echo "$SOURCE_FILES_CHANGED" | grep "$PACKAGE_JSON$")" ] && \
+                       check_package_json_changes; then
+                        echo "Only version field changed in $PACKAGE_JSON, skipping version bump"
+                        SHOULD_BUMP=false
+                        return
+                    fi
+                    
+                    echo "Source files changed:"
+                    echo "$SOURCE_FILES_CHANGED"

---

271-271: Remove trailing spaces.

Remove trailing spaces to maintain consistent code style.

Apply this diff to fix the trailing spaces:

-                DIFF=$(git diff HEAD~1 HEAD -- package.json)
-                
+                DIFF=$(git diff HEAD~1 HEAD -- package.json)
+
                 # Count number of changed lines that aren't the version field
-                NON_VERSION_CHANGES=$(echo "$DIFF" | grep -v '"version":' | grep '^[+-]' | wc -l)
-                
+                NON_VERSION_CHANGES=$(echo "$DIFF" | grep -v '"version":' | grep '^[+-]' | wc -l)
+

Also applies to: 274-274

🧰 Tools

🪛 YAMLlint (1.35.1)

[error] 271-271: trailing spaces

(trailing-spaces)

Configuration used: CodeRabbit UI
Review profile: CHILL
Plan: Pro